{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Validating and Importing User-Item-Interaction Data <a class=\"anchor\" id=\"top\"></a>\n",
    "\n",
    "In this notebook, you will choose a dataset and prepare it for use with Amazon Personalize.\n",
    "\n",
    "1. [Introduction](#intro)\n",
    "1. [Choose a dataset or data source](#source)\n",
    "1. [Prepare your data](#prepare)\n",
    "1. [Create dataset groups and the interactions dataset](#group_dataset)\n",
    "1. [Configure an S3 bucket and an IAM role](#bucket_role)\n",
    "1. [Import the interactions data](#import)\n",
    "\n",
    "## Introduction <a class=\"anchor\" id=\"intro\"></a>\n",
    "\n",
    "For the most part, the algorithms in Amazon Personalize (called recipes) look to solve different tasks, explained here:\n",
    "\n",
    "1. **User Personalization** - New release that supports ALL HRNN workflows / user personalization needs, it will be what we use here.\n",
    "1. **HRNN & HRNN-Metadata** - Recommends items based on previous user interactions with items.\n",
    "1. **HRNN-Coldstart** - Recommends new items for which interaction data is not yet available.\n",
    "1. **Personalized-Ranking** - Takes a collection of items and then orders them in probable order of interest using an HRNN-like approach.\n",
    "1. **SIMS (Similar Items)** - Given one item, recommends other items also interacted with by users.\n",
    "1. **Popularity-Count** - Recommends the most popular items, if HRNN or HRNN-Metadata do not have an answer - this is returned by default.\n",
    "\n",
    "No matter the use case, the algorithms all share a base of learning on user-item-interaction data which is defined by 3 core attributes:\n",
    "\n",
    "1. **UserID** - The user who interacted\n",
    "1. **ItemID** - The item the user interacted with\n",
    "1. **Timestamp** - The time at which the interaction occurred\n",
    "\n",
    "We also support event types and event values defined by:\n",
    "\n",
    "1. **Event Type** - Categorical label of an event (browse, purchased, rated, etc).\n",
    "1. **Event Value** - A value corresponding to the event type that occurred. Generally speaking, we look for normalized values between 0 and 1 over the event types. For example, if there are three phases to complete a transaction (clicked, added-to-cart, and purchased), then there would be an event_value for each phase as 0.33, 0.66, and 1.0 respectfully.\n",
    "\n",
    "The event type and event value fields are additional data which can be used to filter the data sent for training the personalization model. In this particular exercise we will not have an event type or event value. \n",
    "\n",
    "## Choose a dataset or data source <a class=\"anchor\" id=\"source\"></a>\n",
    "[Back to top](#top)\n",
    "\n",
    "As we mentioned, the user-item-iteraction data is key for getting started with the service. This means we need to look for use cases that generate that kind of data, a few common examples are:\n",
    "\n",
    "1. Video-on-demand applications\n",
    "1. E-commerce platforms\n",
    "1. Social media aggregators / platforms\n",
    "\n",
    "There are a few guidelines for scoping a problem suitable for Personalize. We recommend the values below as a starting point, although the [official limits](https://docs.aws.amazon.com/personalize/latest/dg/limits.html) lie a little lower.\n",
    "\n",
    "* Authenticated users\n",
    "* At least 50 unique users\n",
    "* At least 100 unique items\n",
    "* At least 2 dozen interactions for each user \n",
    "\n",
    "Most of the time this is easily attainable, and if you are low in one category, you can often make up for it by having a larger number in another category.\n",
    "\n",
    "Generally speaking your data will not arrive in a perfect form for Personalize, and will take some modification to be structured correctly. This notebook looks to guide you through all of that. \n",
    "\n",
    "To begin with, we are going to use the latest MovieLens dataset, this dataset has over 25 million interactions and a rich collection of metadata for items, there is also a smaller version of this dataset, which can be used to shorten training times, while still incorporating the same capabilities as the full dataset. Set USE_FULL_MOVIELENS to True to use the full dataset."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "![](img/01.png)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "USE_FULL_MOVIELENS = False"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "First, you will download the dataset and unzip it in a new folder using the code below."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "--2020-11-18 10:38:57--  http://files.grouplens.org/datasets/movielens/ml-latest-small.zip\n",
      "Resolving files.grouplens.org (files.grouplens.org)... 128.101.65.152\n",
      "Connecting to files.grouplens.org (files.grouplens.org)|128.101.65.152|:80... connected.\n",
      "HTTP request sent, awaiting response... 200 OK\n",
      "Length: 978202 (955K) [application/zip]\n",
      "Saving to: ‘ml-latest-small.zip’\n",
      "\n",
      "ml-latest-small.zip 100%[===================>] 955.28K  4.14MB/s    in 0.2s    \n",
      "\n",
      "2020-11-18 10:38:58 (4.14 MB/s) - ‘ml-latest-small.zip’ saved [978202/978202]\n",
      "\n",
      "Archive:  ml-latest-small.zip\n",
      "   creating: ml-latest-small/\n",
      "  inflating: ml-latest-small/links.csv  \n",
      "  inflating: ml-latest-small/tags.csv  \n",
      "  inflating: ml-latest-small/ratings.csv  \n",
      "  inflating: ml-latest-small/README.txt  \n",
      "  inflating: ml-latest-small/movies.csv  \n"
     ]
    }
   ],
   "source": [
    "data_dir = \"poc_data\"\n",
    "!mkdir $data_dir\n",
    "\n",
    "if not USE_FULL_MOVIELENS:\n",
    "    !cd $data_dir && wget http://files.grouplens.org/datasets/movielens/ml-latest-small.zip\n",
    "    !cd $data_dir && unzip -o ml-latest-small.zip\n",
    "    dataset_dir = data_dir + \"/ml-latest-small/\"\n",
    "else:\n",
    "    !cd $data_dir && wget http://files.grouplens.org/datasets/movielens/ml-25m.zip\n",
    "    !cd $data_dir && unzip -o ml-25m.zip\n",
    "    dataset_dir = data_dir + \"/ml-25m/\""
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Take a look at the data files you have downloaded."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "links.csv  movies.csv  ratings.csv  README.txt\ttags.csv\r\n"
     ]
    }
   ],
   "source": [
    "!ls $dataset_dir"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "At present not much is known except that we have a few CSVs and a readme. Next we will output the readme to learn more!"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Summary\r\n",
      "=======\r\n",
      "\r\n",
      "This dataset (ml-latest-small) describes 5-star rating and free-text tagging activity from [MovieLens](http://movielens.org), a movie recommendation service. It contains 100836 ratings and 3683 tag applications across 9742 movies. These data were created by 610 users between March 29, 1996 and September 24, 2018. This dataset was generated on September 26, 2018.\r\n",
      "\r\n",
      "Users were selected at random for inclusion. All selected users had rated at least 20 movies. No demographic information is included. Each user is represented by an id, and no other information is provided.\r\n",
      "\r\n",
      "The data are contained in the files `links.csv`, `movies.csv`, `ratings.csv` and `tags.csv`. More details about the contents and use of all these files follows.\r\n",
      "\r\n",
      "This is a *development* dataset. As such, it may change over time and is not an appropriate dataset for shared research results. See available *benchmark* datasets if that is your intent.\r\n",
      "\r\n",
      "This and other GroupLens data sets are publicly available for download at <http://grouplens.org/datasets/>.\r\n",
      "\r\n",
      "\r\n",
      "Usage License\r\n",
      "=============\r\n",
      "\r\n",
      "Neither the University of Minnesota nor any of the researchers involved can guarantee the correctness of the data, its suitability for any particular purpose, or the validity of results based on the use of the data set. The data set may be used for any research purposes under the following conditions:\r\n",
      "\r\n",
      "* The user may not state or imply any endorsement from the University of Minnesota or the GroupLens Research Group.\r\n",
      "* The user must acknowledge the use of the data set in publications resulting from the use of the data set (see below for citation information).\r\n",
      "* The user may redistribute the data set, including transformations, so long as it is distributed under these same license conditions.\r\n",
      "* The user may not use this information for any commercial or revenue-bearing purposes without first obtaining permission from a faculty member of the GroupLens Research Project at the University of Minnesota.\r\n",
      "* The executable software scripts are provided \"as is\" without warranty of any kind, either expressed or implied, including, but not limited to, the implied warranties of merchantability and fitness for a particular purpose. The entire risk as to the quality and performance of them is with you. Should the program prove defective, you assume the cost of all necessary servicing, repair or correction.\r\n",
      "\r\n",
      "In no event shall the University of Minnesota, its affiliates or employees be liable to you for any damages arising out of the use or inability to use these programs (including but not limited to loss of data or data being rendered inaccurate).\r\n",
      "\r\n",
      "If you have any further questions or comments, please email <grouplens-info@umn.edu>\r\n",
      "\r\n",
      "\r\n",
      "Citation\r\n",
      "========\r\n",
      "\r\n",
      "To acknowledge use of the dataset in publications, please cite the following paper:\r\n",
      "\r\n",
      "> F. Maxwell Harper and Joseph A. Konstan. 2015. The MovieLens Datasets: History and Context. ACM Transactions on Interactive Intelligent Systems (TiiS) 5, 4: 19:1–19:19. <https://doi.org/10.1145/2827872>\r\n",
      "\r\n",
      "\r\n",
      "Further Information About GroupLens\r\n",
      "===================================\r\n",
      "\r\n",
      "GroupLens is a research group in the Department of Computer Science and Engineering at the University of Minnesota. Since its inception in 1992, GroupLens's research projects have explored a variety of fields including:\r\n",
      "\r\n",
      "* recommender systems\r\n",
      "* online communities\r\n",
      "* mobile and ubiquitious technologies\r\n",
      "* digital libraries\r\n",
      "* local geographic information systems\r\n",
      "\r\n",
      "GroupLens Research operates a movie recommender based on collaborative filtering, MovieLens, which is the source of these data. We encourage you to visit <http://movielens.org> to try it out! If you have exciting ideas for experimental work to conduct on MovieLens, send us an email at <grouplens-info@cs.umn.edu> - we are always interested in working with external collaborators.\r\n",
      "\r\n",
      "\r\n",
      "Content and Use of Files\r\n",
      "========================\r\n",
      "\r\n",
      "Formatting and Encoding\r\n",
      "-----------------------\r\n",
      "\r\n",
      "The dataset files are written as [comma-separated values](http://en.wikipedia.org/wiki/Comma-separated_values) files with a single header row. Columns that contain commas (`,`) are escaped using double-quotes (`\"`). These files are encoded as UTF-8. If accented characters in movie titles or tag values (e.g. Misérables, Les (1995)) display incorrectly, make sure that any program reading the data, such as a text editor, terminal, or script, is configured for UTF-8.\r\n",
      "\r\n",
      "\r\n",
      "User Ids\r\n",
      "--------\r\n",
      "\r\n",
      "MovieLens users were selected at random for inclusion. Their ids have been anonymized. User ids are consistent between `ratings.csv` and `tags.csv` (i.e., the same id refers to the same user across the two files).\r\n",
      "\r\n",
      "\r\n",
      "Movie Ids\r\n",
      "---------\r\n",
      "\r\n",
      "Only movies with at least one rating or tag are included in the dataset. These movie ids are consistent with those used on the MovieLens web site (e.g., id `1` corresponds to the URL <https://movielens.org/movies/1>). Movie ids are consistent between `ratings.csv`, `tags.csv`, `movies.csv`, and `links.csv` (i.e., the same id refers to the same movie across these four data files).\r\n",
      "\r\n",
      "\r\n",
      "Ratings Data File Structure (ratings.csv)\r\n",
      "-----------------------------------------\r\n",
      "\r\n",
      "All ratings are contained in the file `ratings.csv`. Each line of this file after the header row represents one rating of one movie by one user, and has the following format:\r\n",
      "\r\n",
      "    userId,movieId,rating,timestamp\r\n",
      "\r\n",
      "The lines within this file are ordered first by userId, then, within user, by movieId.\r\n",
      "\r\n",
      "Ratings are made on a 5-star scale, with half-star increments (0.5 stars - 5.0 stars).\r\n",
      "\r\n",
      "Timestamps represent seconds since midnight Coordinated Universal Time (UTC) of January 1, 1970.\r\n",
      "\r\n",
      "\r\n",
      "Tags Data File Structure (tags.csv)\r\n",
      "-----------------------------------\r\n",
      "\r\n",
      "All tags are contained in the file `tags.csv`. Each line of this file after the header row represents one tag applied to one movie by one user, and has the following format:\r\n",
      "\r\n",
      "    userId,movieId,tag,timestamp\r\n",
      "\r\n",
      "The lines within this file are ordered first by userId, then, within user, by movieId.\r\n",
      "\r\n",
      "Tags are user-generated metadata about movies. Each tag is typically a single word or short phrase. The meaning, value, and purpose of a particular tag is determined by each user.\r\n",
      "\r\n",
      "Timestamps represent seconds since midnight Coordinated Universal Time (UTC) of January 1, 1970.\r\n",
      "\r\n",
      "\r\n",
      "Movies Data File Structure (movies.csv)\r\n",
      "---------------------------------------\r\n",
      "\r\n",
      "Movie information is contained in the file `movies.csv`. Each line of this file after the header row represents one movie, and has the following format:\r\n",
      "\r\n",
      "    movieId,title,genres\r\n",
      "\r\n",
      "Movie titles are entered manually or imported from <https://www.themoviedb.org/>, and include the year of release in parentheses. Errors and inconsistencies may exist in these titles.\r\n",
      "\r\n",
      "Genres are a pipe-separated list, and are selected from the following:\r\n",
      "\r\n",
      "* Action\r\n",
      "* Adventure\r\n",
      "* Animation\r\n",
      "* Children's\r\n",
      "* Comedy\r\n",
      "* Crime\r\n",
      "* Documentary\r\n",
      "* Drama\r\n",
      "* Fantasy\r\n",
      "* Film-Noir\r\n",
      "* Horror\r\n",
      "* Musical\r\n",
      "* Mystery\r\n",
      "* Romance\r\n",
      "* Sci-Fi\r\n",
      "* Thriller\r\n",
      "* War\r\n",
      "* Western\r\n",
      "* (no genres listed)\r\n",
      "\r\n",
      "\r\n",
      "Links Data File Structure (links.csv)\r\n",
      "---------------------------------------\r\n",
      "\r\n",
      "Identifiers that can be used to link to other sources of movie data are contained in the file `links.csv`. Each line of this file after the header row represents one movie, and has the following format:\r\n",
      "\r\n",
      "    movieId,imdbId,tmdbId\r\n",
      "\r\n",
      "movieId is an identifier for movies used by <https://movielens.org>. E.g., the movie Toy Story has the link <https://movielens.org/movies/1>.\r\n",
      "\r\n",
      "imdbId is an identifier for movies used by <http://www.imdb.com>. E.g., the movie Toy Story has the link <http://www.imdb.com/title/tt0114709/>.\r\n",
      "\r\n",
      "tmdbId is an identifier for movies used by <https://www.themoviedb.org>. E.g., the movie Toy Story has the link <https://www.themoviedb.org/movie/862>.\r\n",
      "\r\n",
      "Use of the resources listed above is subject to the terms of each provider.\r\n",
      "\r\n",
      "\r\n",
      "Cross-Validation\r\n",
      "----------------\r\n",
      "\r\n",
      "Prior versions of the MovieLens dataset included either pre-computed cross-folds or scripts to perform this computation. We no longer bundle either of these features with the dataset, since most modern toolkits provide this as a built-in feature. If you wish to learn about standard approaches to cross-fold computation in the context of recommender systems evaluation, see [LensKit](http://lenskit.org) for tools, documentation, and open-source code examples.\r\n"
     ]
    }
   ],
   "source": [
    "!pygmentize $dataset_dir/README.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From the README, we see there is a file `ratings.csv` that should work as a proxy for our interactions data, after all rating a film definitely is a form of interacting with it. The dataset also has some genre information as some movie genome data. In this POC we will focus on the interactions and the genre data.\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Prepare your data <a class=\"anchor\" id=\"prepare\"></a>\n",
    "[Back to top](#top)\n",
    "\n",
    "The next thing to be done is to load the data and confirm the data is in a good state, then save it to a CSV where it is ready to be used with Amazon Personalize.\n",
    "\n",
    "To get started, import a collection of Python libraries commonly used in data science."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "from time import sleep\n",
    "import json\n",
    "from datetime import datetime\n",
    "import boto3\n",
    "import pandas as pd"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next,open the data file and take a look at the first several rows."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>4.0</td>\n",
       "      <td>964982703</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>4.0</td>\n",
       "      <td>964981247</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>4.0</td>\n",
       "      <td>964982224</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>47</td>\n",
       "      <td>5.0</td>\n",
       "      <td>964983815</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>5.0</td>\n",
       "      <td>964982931</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  movieId  rating  timestamp\n",
       "0       1        1     4.0  964982703\n",
       "1       1        3     4.0  964981247\n",
       "2       1        6     4.0  964982224\n",
       "3       1       47     5.0  964983815\n",
       "4       1       50     5.0  964982931"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "original_data = pd.read_csv(dataset_dir + '/ratings.csv')\n",
    "original_data.head(5)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "(100836, 4)"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "original_data.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>rating</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>100836.000000</td>\n",
       "      <td>100836.000000</td>\n",
       "      <td>100836.000000</td>\n",
       "      <td>1.008360e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>326.127564</td>\n",
       "      <td>19435.295718</td>\n",
       "      <td>3.501557</td>\n",
       "      <td>1.205946e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>182.618491</td>\n",
       "      <td>35530.987199</td>\n",
       "      <td>1.042529</td>\n",
       "      <td>2.162610e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>0.500000</td>\n",
       "      <td>8.281246e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>177.000000</td>\n",
       "      <td>1199.000000</td>\n",
       "      <td>3.000000</td>\n",
       "      <td>1.019124e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>325.000000</td>\n",
       "      <td>2991.000000</td>\n",
       "      <td>3.500000</td>\n",
       "      <td>1.186087e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>477.000000</td>\n",
       "      <td>8122.000000</td>\n",
       "      <td>4.000000</td>\n",
       "      <td>1.435994e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>610.000000</td>\n",
       "      <td>193609.000000</td>\n",
       "      <td>5.000000</td>\n",
       "      <td>1.537799e+09</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              userId        movieId         rating     timestamp\n",
       "count  100836.000000  100836.000000  100836.000000  1.008360e+05\n",
       "mean      326.127564   19435.295718       3.501557  1.205946e+09\n",
       "std       182.618491   35530.987199       1.042529  2.162610e+08\n",
       "min         1.000000       1.000000       0.500000  8.281246e+08\n",
       "25%       177.000000    1199.000000       3.000000  1.019124e+09\n",
       "50%       325.000000    2991.000000       3.500000  1.186087e+09\n",
       "75%       477.000000    8122.000000       4.000000  1.435994e+09\n",
       "max       610.000000  193609.000000       5.000000  1.537799e+09"
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "original_data.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This shows that we have a good range of values for `userId` and `movieId`. Next, it is always a good idea to confirm the data format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "RangeIndex: 100836 entries, 0 to 100835\n",
      "Data columns (total 4 columns):\n",
      " #   Column     Non-Null Count   Dtype  \n",
      "---  ------     --------------   -----  \n",
      " 0   userId     100836 non-null  int64  \n",
      " 1   movieId    100836 non-null  int64  \n",
      " 2   rating     100836 non-null  float64\n",
      " 3   timestamp  100836 non-null  int64  \n",
      "dtypes: float64(1), int64(3)\n",
      "memory usage: 3.1 MB\n"
     ]
    }
   ],
   "source": [
    "original_data.info()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "userId       False\n",
       "movieId      False\n",
       "rating       False\n",
       "timestamp    False\n",
       "dtype: bool"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "original_data.isnull().any()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "From this, you can see that there are a total of (25,000,095 for full 100836 for small) entries in the dataset, with 4 columns, and each cell stored as int64 format, with the exception of the rating whihch is a float64.\n",
    "\n",
    "The int64 format is clearly suitable for `userId` and `movieId`. However, we need to diver deeper to understand the timestamps in the data. To use Amazon Personalize, you need to save timestamps in [Unix Epoch](https://en.wikipedia.org/wiki/Unix_time) format.\n",
    "\n",
    "Currently, the timestamp values are not human-readable. So let's grab an arbitrary timestamp value and figure out how to interpret it."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Do a quick sanity check on the transformed dataset by picking an arbitrary timestamp and transforming it to a human-readable format."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "964982681.0\n",
      "2000-07-30 18:44:41\n"
     ]
    }
   ],
   "source": [
    "arb_time_stamp = original_data.iloc[50]['timestamp']\n",
    "print(arb_time_stamp)\n",
    "print(datetime.utcfromtimestamp(arb_time_stamp).strftime('%Y-%m-%d %H:%M:%S'))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "This date makes sense as a timestamp, so we can continue formatting the rest of the data. Remember, the data we need is user-item-interaction data, which is `userId`, `movieId`, and `timestamp` in this case. Our dataset has an additional column, `rating`, which can be dropped from the dataset after we have leveraged it to focus on positive interactions."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since this is an explicit feedback movie rating dataset, it includes movies rated from 1 to 5, we want to include only moves that weree \"liked\" by the users, and simulate a implicit dataset that is more like what data would be gathered by a VOD platform. For that so we will filter out all interactions under 2 out of 5, and create an EVENT_Type of \"click\" and an EVENT_Type of \"watch\". We will then assign all movies rated 2 and above as \"click\" and movies above 4 and above as \"click\" and \"watch\".\n",
    "\n",
    "Note this is to correspond with the events we are modeling, for a real data set you would actually model based on implicit feedback like clicks, watches and/or explicit feedback like ratings, likes etc."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>EVENT_TYPE</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>964982703</td>\n",
       "      <td>watch</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>964981247</td>\n",
       "      <td>watch</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>964982224</td>\n",
       "      <td>watch</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>47</td>\n",
       "      <td>964983815</td>\n",
       "      <td>watch</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>964982931</td>\n",
       "      <td>watch</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  movieId  timestamp EVENT_TYPE\n",
       "0       1        1  964982703      watch\n",
       "1       1        3  964981247      watch\n",
       "2       1        6  964982224      watch\n",
       "3       1       47  964983815      watch\n",
       "4       1       50  964982931      watch"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "watched_df = original_data.copy()\n",
    "watched_df = watched_df[watched_df['rating'] > 3]\n",
    "watched_df = watched_df[['userId', 'movieId', 'timestamp']]\n",
    "watched_df['EVENT_TYPE']='watch'\n",
    "watched_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>timestamp</th>\n",
       "      <th>EVENT_TYPE</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>1</td>\n",
       "      <td>1</td>\n",
       "      <td>964982703</td>\n",
       "      <td>click</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>1</td>\n",
       "      <td>3</td>\n",
       "      <td>964981247</td>\n",
       "      <td>click</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>1</td>\n",
       "      <td>6</td>\n",
       "      <td>964982224</td>\n",
       "      <td>click</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>1</td>\n",
       "      <td>47</td>\n",
       "      <td>964983815</td>\n",
       "      <td>click</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>1</td>\n",
       "      <td>50</td>\n",
       "      <td>964982931</td>\n",
       "      <td>click</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "   userId  movieId  timestamp EVENT_TYPE\n",
       "0       1        1  964982703      click\n",
       "1       1        3  964981247      click\n",
       "2       1        6  964982224      click\n",
       "3       1       47  964983815      click\n",
       "4       1       50  964982931      click"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "clicked_df = original_data.copy()\n",
    "clicked_df = clicked_df[clicked_df['rating'] > 1]\n",
    "clicked_df = clicked_df[['userId', 'movieId', 'timestamp']]\n",
    "clicked_df['EVENT_TYPE']='click'\n",
    "clicked_df.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [],
   "source": [
    "interactions_df = clicked_df.copy()\n",
    "interactions_df = interactions_df.append(watched_df)\n",
    "interactions_df.sort_values(\"timestamp\", axis = 0, ascending = True, \n",
    "                 inplace = True, na_position ='last') "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<class 'pandas.core.frame.DataFrame'>\n",
      "Int64Index: 158371 entries, 66679 to 81092\n",
      "Data columns (total 4 columns):\n",
      " #   Column      Non-Null Count   Dtype \n",
      "---  ------      --------------   ----- \n",
      " 0   userId      158371 non-null  int64 \n",
      " 1   movieId     158371 non-null  int64 \n",
      " 2   timestamp   158371 non-null  int64 \n",
      " 3   EVENT_TYPE  158371 non-null  object\n",
      "dtypes: int64(3), object(1)\n",
      "memory usage: 6.0+ MB\n"
     ]
    }
   ],
   "source": [
    "interactions_df.info()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "lets look at what the new dataset looks like "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 16,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>userId</th>\n",
       "      <th>movieId</th>\n",
       "      <th>timestamp</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>count</th>\n",
       "      <td>158371.000000</td>\n",
       "      <td>158371.000000</td>\n",
       "      <td>1.583710e+05</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>mean</th>\n",
       "      <td>323.940804</td>\n",
       "      <td>19869.485146</td>\n",
       "      <td>1.211627e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>std</th>\n",
       "      <td>182.195975</td>\n",
       "      <td>35809.058185</td>\n",
       "      <td>2.137376e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>min</th>\n",
       "      <td>1.000000</td>\n",
       "      <td>1.000000</td>\n",
       "      <td>8.281246e+08</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>25%</th>\n",
       "      <td>175.000000</td>\n",
       "      <td>1203.000000</td>\n",
       "      <td>1.031072e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>50%</th>\n",
       "      <td>325.000000</td>\n",
       "      <td>3033.000000</td>\n",
       "      <td>1.193289e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>75%</th>\n",
       "      <td>475.000000</td>\n",
       "      <td>8528.000000</td>\n",
       "      <td>1.435998e+09</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>max</th>\n",
       "      <td>610.000000</td>\n",
       "      <td>193609.000000</td>\n",
       "      <td>1.537799e+09</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "              userId        movieId     timestamp\n",
       "count  158371.000000  158371.000000  1.583710e+05\n",
       "mean      323.940804   19869.485146  1.211627e+09\n",
       "std       182.195975   35809.058185  2.137376e+08\n",
       "min         1.000000       1.000000  8.281246e+08\n",
       "25%       175.000000    1203.000000  1.031072e+09\n",
       "50%       325.000000    3033.000000  1.193289e+09\n",
       "75%       475.000000    8528.000000  1.435998e+09\n",
       "max       610.000000  193609.000000  1.537799e+09"
      ]
     },
     "execution_count": 16,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "interactions_df.describe()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "After manipulating the data, always confirm if the data format has changed."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 17,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "userId         int64\n",
       "movieId        int64\n",
       "timestamp      int64\n",
       "EVENT_TYPE    object\n",
       "dtype: object"
      ]
     },
     "execution_count": 17,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "interactions_df.dtypes"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    " Amazon Personalize has default column names for users, items, and timestamp. These default column names are `USER_ID`, `ITEM_ID`, AND `TIMESTAMP`. So the final modification to the dataset is to replace the existing column headers with the default headers."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "metadata": {},
   "outputs": [],
   "source": [
    "interactions_df.rename(columns = {'userId':'USER_ID', 'movieId':'ITEM_ID', \n",
    "                              'timestamp':'TIMESTAMP'}, inplace = True) "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "That's it! At this point the data is ready to go, and we just need to save it as a CSV file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "metadata": {},
   "outputs": [],
   "source": [
    "interactions_filename = \"interactions.csv\"\n",
    "interactions_df.to_csv((data_dir+\"/\"+interactions_filename), index=False, float_format='%.0f')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Create dataset groups and the interactions dataset <a class=\"anchor\" id=\"group_dataset\"></a>\n",
    "[Back to top](#top)\n",
    "\n",
    "The highest level of isolation and abstraction with Amazon Personalize is a *dataset group*. Information stored within one of these dataset groups has no impact on any other dataset group or models created from one - they are completely isolated. This allows you to run many experiments and is part of how we keep your models private and fully trained only on your data. \n",
    "\n",
    "Before importing the data prepared earlier, there needs to be a dataset group and a dataset added to it that handles the interactions.\n",
    "\n",
    "Dataset groups can house the following types of information:\n",
    "\n",
    "* User-item-interactions\n",
    "* Event streams (real-time interactions)\n",
    "* User metadata\n",
    "* Item metadata\n",
    "\n",
    "Before we create the dataset group and the dataset for our interaction data, let's validate that your environment can communicate successfully with Amazon Personalize."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 20,
   "metadata": {},
   "outputs": [],
   "source": [
    "# Configure the SDK to Personalize:\n",
    "personalize = boto3.client('personalize')\n",
    "personalize_runtime = boto3.client('personalize-runtime')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the dataset group\n",
    "\n",
    "The following cell will create a new dataset group with the name `personalize-poc-lastfm`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"datasetGroupArn\": \"arn:aws:personalize:us-east-1:001723200916:dataset-group/personalize-poc-movielens25m\",\n",
      "  \"ResponseMetadata\": {\n",
      "    \"RequestId\": \"850bad28-40c4-4181-864b-3bf367addd5f\",\n",
      "    \"HTTPStatusCode\": 200,\n",
      "    \"HTTPHeaders\": {\n",
      "      \"content-type\": \"application/x-amz-json-1.1\",\n",
      "      \"date\": \"Wed, 18 Nov 2020 10:39:01 GMT\",\n",
      "      \"x-amzn-requestid\": \"850bad28-40c4-4181-864b-3bf367addd5f\",\n",
      "      \"content-length\": \"107\",\n",
      "      \"connection\": \"keep-alive\"\n",
      "    },\n",
      "    \"RetryAttempts\": 0\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "create_dataset_group_response = personalize.create_dataset_group(\n",
    "    name = \"personalize-poc-movielens25m\"\n",
    ")\n",
    "\n",
    "dataset_group_arn = create_dataset_group_response['datasetGroupArn']\n",
    "print(json.dumps(create_dataset_group_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we can use the dataset group, it must be active. This can take a minute or two. Execute the cell below and wait for it to show the ACTIVE status. It checks the status of the dataset group every second, up to a maximum of 3 hours."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DatasetGroup: CREATE PENDING\n",
      "DatasetGroup: ACTIVE\n"
     ]
    }
   ],
   "source": [
    "max_time = time.time() + 3*60*60 # 3 hours\n",
    "while time.time() < max_time:\n",
    "    describe_dataset_group_response = personalize.describe_dataset_group(\n",
    "        datasetGroupArn = dataset_group_arn\n",
    "    )\n",
    "    status = describe_dataset_group_response[\"datasetGroup\"][\"status\"]\n",
    "    print(\"DatasetGroup: {}\".format(status))\n",
    "    \n",
    "    if status == \"ACTIVE\" or status == \"CREATE FAILED\":\n",
    "        break\n",
    "        \n",
    "    time.sleep(60)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now that you have a dataset group, you can create a dataset for the interaction data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create the dataset\n",
    "\n",
    "First, define a schema to tell Amazon Personalize what type of dataset you are uploading. There are several reserved and mandatory keywords required in the schema, based on the type of dataset. More detailed information can be found in the [documentation](https://docs.aws.amazon.com/personalize/latest/dg/how-it-works-dataset-schema.html).\n",
    "\n",
    "Here, you will create a schema for interactions data, which needs the `USER_ID`, `ITEM_ID`, and `TIMESTAMP` fields. These must be defined in the same order in the schema as they appear in the dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"schemaArn\": \"arn:aws:personalize:us-east-1:001723200916:schema/personalize-poc-movielens-interactions\",\n",
      "  \"ResponseMetadata\": {\n",
      "    \"RequestId\": \"36bca0fa-a24a-4841-9674-81fd286806a7\",\n",
      "    \"HTTPStatusCode\": 200,\n",
      "    \"HTTPHeaders\": {\n",
      "      \"content-type\": \"application/x-amz-json-1.1\",\n",
      "      \"date\": \"Wed, 18 Nov 2020 10:40:01 GMT\",\n",
      "      \"x-amzn-requestid\": \"36bca0fa-a24a-4841-9674-81fd286806a7\",\n",
      "      \"content-length\": \"104\",\n",
      "      \"connection\": \"keep-alive\"\n",
      "    },\n",
      "    \"RetryAttempts\": 0\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "interactions_schema = schema = {\n",
    "    \"type\": \"record\",\n",
    "    \"name\": \"Interactions\",\n",
    "    \"namespace\": \"com.amazonaws.personalize.schema\",\n",
    "    \"fields\": [\n",
    "        {\n",
    "            \"name\": \"USER_ID\",\n",
    "            \"type\": \"string\"\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"ITEM_ID\",\n",
    "            \"type\": \"string\"\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"EVENT_TYPE\",\n",
    "            \"type\": \"string\"\n",
    "        },\n",
    "        {\n",
    "            \"name\": \"TIMESTAMP\",\n",
    "            \"type\": \"long\"\n",
    "        }\n",
    "    ],\n",
    "    \"version\": \"1.0\"\n",
    "}\n",
    "\n",
    "create_schema_response = personalize.create_schema(\n",
    "    name = \"personalize-poc-movielens-interactions\",\n",
    "    schema = json.dumps(interactions_schema)\n",
    ")\n",
    "\n",
    "schema_arn = create_schema_response['schemaArn']\n",
    "print(json.dumps(create_schema_response, indent=2))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {},
   "outputs": [],
   "source": [
    "# personalize.list_schemas()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [],
   "source": [
    "# personalize.delete_schema(\n",
    "#     schemaArn='arn:aws:personalize:us-east-1:835319576252:schema/personalize-demo-schema'\n",
    "# #     name = \"personalize-poc-movielens-interactions\",\n",
    "# # #    schema = json.dumps(interactions_schema)\n",
    "# )\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "With a schema created, you can create a dataset within the dataset group. Note, this does not load the data yet. This will happen a few steps later."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"datasetArn\": \"arn:aws:personalize:us-east-1:001723200916:dataset/personalize-poc-movielens25m/INTERACTIONS\",\n",
      "  \"ResponseMetadata\": {\n",
      "    \"RequestId\": \"bdfd9830-5ad0-4031-8c6a-71dcb8f2573b\",\n",
      "    \"HTTPStatusCode\": 200,\n",
      "    \"HTTPHeaders\": {\n",
      "      \"content-type\": \"application/x-amz-json-1.1\",\n",
      "      \"date\": \"Wed, 18 Nov 2020 10:40:02 GMT\",\n",
      "      \"x-amzn-requestid\": \"bdfd9830-5ad0-4031-8c6a-71dcb8f2573b\",\n",
      "      \"content-length\": \"109\",\n",
      "      \"connection\": \"keep-alive\"\n",
      "    },\n",
      "    \"RetryAttempts\": 0\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "dataset_type = \"INTERACTIONS\"\n",
    "create_dataset_response = personalize.create_dataset(\n",
    "    name = \"personalize-poc-movielens-ints\",\n",
    "    datasetType = dataset_type,\n",
    "    datasetGroupArn = dataset_group_arn,\n",
    "    schemaArn = schema_arn\n",
    ")\n",
    "\n",
    "interactions_dataset_arn = create_dataset_response['datasetArn']\n",
    "print(json.dumps(create_dataset_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Configure an S3 bucket and an IAM  role <a class=\"anchor\" id=\"bucket_role\"></a>\n",
    "[Back to top](#top)\n",
    "\n",
    "So far, we have downloaded, manipulated, and saved the data onto the Amazon EBS instance attached to instance running this Jupyter notebook. However, Amazon Personalize will need an S3 bucket to act as the source of your data, as well as IAM roles for accessing that bucket. Let's set all of that up.\n",
    "\n",
    "Use the metadata stored on the instance underlying this Amazon SageMaker notebook, to determine the region it is operating in. If you are using a Jupyter notebook outside of Amazon SageMaker, simply define the region as a string below. The Amazon S3 bucket needs to be in the same region as the Amazon Personalize resources we have been creating so far."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "us-east-1\n"
     ]
    }
   ],
   "source": [
    "with open('/opt/ml/metadata/resource-metadata.json') as notebook_info:\n",
    "    data = json.load(notebook_info)\n",
    "    resource_arn = data['ResourceArn']\n",
    "    region = resource_arn.split(':')[3]\n",
    "print(region)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Amazon S3 bucket names are globally unique. To create a unique bucket name, the code below will append the string `personalizepocvod` to your AWS account number. Then it creates a bucket with this name in the region discovered in the previous cell."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "001723200916personalizepocvod\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{'ResponseMetadata': {'RequestId': '72E4385C116ED35A',\n",
       "  'HostId': 'fk13Pw/ytR4hm2pvhS56i2RSAFeKPQWk+UzSeUqlcSRI/9CgmXx3c0B8QsxGD1YQJ3cFfSx8H9E=',\n",
       "  'HTTPStatusCode': 200,\n",
       "  'HTTPHeaders': {'x-amz-id-2': 'fk13Pw/ytR4hm2pvhS56i2RSAFeKPQWk+UzSeUqlcSRI/9CgmXx3c0B8QsxGD1YQJ3cFfSx8H9E=',\n",
       "   'x-amz-request-id': '72E4385C116ED35A',\n",
       "   'date': 'Wed, 18 Nov 2020 10:40:03 GMT',\n",
       "   'location': '/001723200916personalizepocvod',\n",
       "   'content-length': '0',\n",
       "   'server': 'AmazonS3'},\n",
       "  'RetryAttempts': 0},\n",
       " 'Location': '/001723200916personalizepocvod'}"
      ]
     },
     "execution_count": 28,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s3 = boto3.client('s3')\n",
    "account_id = boto3.client('sts').get_caller_identity().get('Account')\n",
    "bucket_name = account_id + \"personalizepocvod\"\n",
    "print(bucket_name)\n",
    "s3.create_bucket(Bucket=bucket_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Upload data to S3\n",
    "\n",
    "Now that your Amazon S3 bucket has been created, upload the CSV file of our user-item-interaction data. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [],
   "source": [
    "interactions_file_path = data_dir + \"/\" + interactions_filename\n",
    "boto3.Session().resource('s3').Bucket(bucket_name).Object(interactions_filename).upload_file(interactions_file_path)\n",
    "interactions_s3DataPath = \"s3://\"+bucket_name+\"/\"+interactions_filename"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Set the S3 bucket policy\n",
    "Amazon Personalize needs to be able to read the contents of your S3 bucket. So add a bucket policy which allows that."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "{'ResponseMetadata': {'RequestId': '91CF84652BEC84A7',\n",
       "  'HostId': 'R98EWua7XlTrI8agsAky1ZgzFt4QYZzeE7os5phf2QUCxaH2nXp6aJB3Ebn1XQvX0yXv+Jgdl2E=',\n",
       "  'HTTPStatusCode': 204,\n",
       "  'HTTPHeaders': {'x-amz-id-2': 'R98EWua7XlTrI8agsAky1ZgzFt4QYZzeE7os5phf2QUCxaH2nXp6aJB3Ebn1XQvX0yXv+Jgdl2E=',\n",
       "   'x-amz-request-id': '91CF84652BEC84A7',\n",
       "   'date': 'Wed, 18 Nov 2020 10:40:05 GMT',\n",
       "   'server': 'AmazonS3'},\n",
       "  'RetryAttempts': 0}}"
      ]
     },
     "execution_count": 30,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "policy = {\n",
    "    \"Version\": \"2012-10-17\",\n",
    "    \"Id\": \"PersonalizeS3BucketAccessPolicy\",\n",
    "    \"Statement\": [\n",
    "        {\n",
    "            \"Sid\": \"PersonalizeS3BucketAccessPolicy\",\n",
    "            \"Effect\": \"Allow\",\n",
    "            \"Principal\": {\n",
    "                \"Service\": \"personalize.amazonaws.com\"\n",
    "            },\n",
    "            \"Action\": [\n",
    "                \"s3:*Object\",\n",
    "                \"s3:ListBucket\"\n",
    "            ],\n",
    "            \"Resource\": [\n",
    "                \"arn:aws:s3:::{}\".format(bucket_name),\n",
    "                \"arn:aws:s3:::{}/*\".format(bucket_name)\n",
    "            ]\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "\n",
    "s3.put_bucket_policy(Bucket=bucket_name, Policy=json.dumps(policy))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Create an IAM role\n",
    "\n",
    "Amazon Personalize needs the ability to assume roles in AWS in order to have the permissions to execute certain tasks. Let's create an IAM role and attach the required policies to it. The code below attaches very permissive policies; please use more restrictive policies for any production application."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "arn:aws:iam::001723200916:role/PersonalizeRolePOC\n"
     ]
    }
   ],
   "source": [
    "iam = boto3.client(\"iam\")\n",
    "\n",
    "role_name = \"PersonalizeRolePOC\"\n",
    "assume_role_policy_document = {\n",
    "    \"Version\": \"2012-10-17\",\n",
    "    \"Statement\": [\n",
    "        {\n",
    "          \"Effect\": \"Allow\",\n",
    "          \"Principal\": {\n",
    "            \"Service\": \"personalize.amazonaws.com\"\n",
    "          },\n",
    "          \"Action\": \"sts:AssumeRole\"\n",
    "        }\n",
    "    ]\n",
    "}\n",
    "\n",
    "create_role_response = iam.create_role(\n",
    "    RoleName = role_name,\n",
    "    AssumeRolePolicyDocument = json.dumps(assume_role_policy_document)\n",
    ")\n",
    "\n",
    "# AmazonPersonalizeFullAccess provides access to any S3 bucket with a name that includes \"personalize\" or \"Personalize\" \n",
    "# if you would like to use a bucket with a different name, please consider creating and attaching a new policy\n",
    "# that provides read access to your bucket or attaching the AmazonS3ReadOnlyAccess policy to the role\n",
    "policy_arn = \"arn:aws:iam::aws:policy/service-role/AmazonPersonalizeFullAccess\"\n",
    "iam.attach_role_policy(\n",
    "    RoleName = role_name,\n",
    "    PolicyArn = policy_arn\n",
    ")\n",
    "\n",
    "# Now add S3 support\n",
    "iam.attach_role_policy(\n",
    "    PolicyArn='arn:aws:iam::aws:policy/AmazonS3FullAccess',\n",
    "    RoleName=role_name\n",
    ")\n",
    "time.sleep(60) # wait for a minute to allow IAM role policy attachment to propagate\n",
    "\n",
    "role_arn = create_role_response[\"Role\"][\"Arn\"]\n",
    "print(role_arn)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Import the interactions data <a class=\"anchor\" id=\"import\"></a>\n",
    "[Back to top](#top)\n",
    "\n",
    "Earlier you created the dataset group and dataset to house your information, so now you will execute an import job that will load the data from the S3 bucket into the Amazon Personalize dataset. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "{\n",
      "  \"datasetImportJobArn\": \"arn:aws:personalize:us-east-1:001723200916:dataset-import-job/personalize-poc-import1\",\n",
      "  \"ResponseMetadata\": {\n",
      "    \"RequestId\": \"90524d37-7595-41ca-8baa-f0d9489370ec\",\n",
      "    \"HTTPStatusCode\": 200,\n",
      "    \"HTTPHeaders\": {\n",
      "      \"content-type\": \"application/x-amz-json-1.1\",\n",
      "      \"date\": \"Wed, 18 Nov 2020 10:41:04 GMT\",\n",
      "      \"x-amzn-requestid\": \"90524d37-7595-41ca-8baa-f0d9489370ec\",\n",
      "      \"content-length\": \"111\",\n",
      "      \"connection\": \"keep-alive\"\n",
      "    },\n",
      "    \"RetryAttempts\": 0\n",
      "  }\n",
      "}\n"
     ]
    }
   ],
   "source": [
    "create_dataset_import_job_response = personalize.create_dataset_import_job(\n",
    "    jobName = \"personalize-poc-import1\",\n",
    "    datasetArn = interactions_dataset_arn,\n",
    "    dataSource = {\n",
    "        \"dataLocation\": \"s3://{}/{}\".format(bucket_name, interactions_filename)\n",
    "    },\n",
    "    roleArn = role_arn\n",
    ")\n",
    "\n",
    "dataset_import_job_arn = create_dataset_import_job_response['datasetImportJobArn']\n",
    "print(json.dumps(create_dataset_import_job_response, indent=2))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Before we can use the dataset, the import job must be active. Execute the cell below and wait for it to show the ACTIVE status. It checks the status of the import job every second, up to a maximum of 6 hours.\n",
    "\n",
    "Importing the data can take some time, depending on the size of the dataset. In this workshop, the data import job should take around 15 minutes."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "DatasetImportJob: CREATE PENDING\n",
      "DatasetImportJob: CREATE IN_PROGRESS\n",
      "DatasetImportJob: CREATE IN_PROGRESS\n",
      "DatasetImportJob: CREATE IN_PROGRESS\n",
      "DatasetImportJob: CREATE IN_PROGRESS\n",
      "DatasetImportJob: CREATE IN_PROGRESS\n",
      "DatasetImportJob: CREATE IN_PROGRESS\n",
      "DatasetImportJob: CREATE IN_PROGRESS\n"
     ]
    }
   ],
   "source": [
    "%%time\n",
    "\n",
    "max_time = time.time() + 6*60*60 # 6 hours\n",
    "while time.time() < max_time:\n",
    "    describe_dataset_import_job_response = personalize.describe_dataset_import_job(\n",
    "        datasetImportJobArn = dataset_import_job_arn\n",
    "    )\n",
    "    status = describe_dataset_import_job_response[\"datasetImportJob\"]['status']\n",
    "    print(\"DatasetImportJob: {}\".format(status))\n",
    "    \n",
    "    if status == \"ACTIVE\" or status == \"CREATE FAILED\":\n",
    "        break\n",
    "        \n",
    "    time.sleep(60)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "When the dataset import is active, you are ready to start building models with SIMS, Personalized-Ranking, Popularity-Count, and User Personalization. This process will continue in other notebooks. Run the cell below before moving on to store a few values for usage in the next notebooks."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store USE_FULL_MOVIELENS\n",
    "%store dataset_dir\n",
    "%store interactions_dataset_arn\n",
    "%store dataset_group_arn\n",
    "%store bucket_name\n",
    "%store role_arn\n",
    "%store role_name\n",
    "%store data_dir\n",
    "%store region"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Release Resources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%javascript\n",
    "Jupyter.notebook.save_checkpoint();\n",
    "Jupyter.notebook.session.delete();"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "conda_python3",
   "language": "python",
   "name": "conda_python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
